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Executive Summary 


This report is part of a long-term collaboration of the Quality Teaching for English Learners initiative at 
WestEd with the Fort Worth Independent School District. The objective of this report is to explore how 
ninth grade data can predict the graduation outcomes of newcomers. Based upon predictor variables, we 
construct binary indicators for students "at risk" of not graduating. We compare the utility of English 
language proficiency measures and attendance in the ninth grade year for predicting graduation. 


Key Findings 

e Higher ninth grade TELPAS Speaking and Listening ratings were associated with higher graduation 
rates. For example, students rated as at the Beginning level on Speaking graduated at a rate of 46%, 
compared to 61% among students rated at Advanced or higher. These differences were similar in 
the Listening domain, with students rated Beginning less likely to graduate (46%) than students 
rated Advanced or higher (60%). These associations were not statistically significant. 

e The association between TELPAS Writing and graduation was both stronger and statistically 
significant. Students rated as Advanced or higher in Writing were much more likely to graduate 
(65%) than students rated Beginning (41%). The TELPAS Reading score and attendance (measured 
as days absent) were statistically significant predictors of the graduation outcome. That is, there is a 
relationship between these predictor variables and the probability of students graduating. 

e Creating indicators to identify students who are "at risk" of not graduating must simultaneously 
identify as many non-graduates as possible while identifying a group that actually had a low 
graduation rate. 

e For example, if a score lower than 650 on the TELPAS reading is used to identify students at risk of 
not graduating, 50% of all non-graduates will be identified, but 40% of "at-risk" students will 
actually graduate. 

e Alternatively, if students absent at least five days are identified as "at risk", 83% of all non-graduates 
will be identified. The graduation rate of students with this many absences, however, is 45%. 
Because this rate is close to the overall graduation rate of 47%, the students identified by this 
indicator were not actually much less likely to graduate. 

e Combining predictor variables may yield better indicators. For instance, one way to define "at-risk" 
is as:1) a TELPAS Reading score of less than 646, OR 2) more than eight days absent. This at-risk 
indicator definition would identify 70% of all non-graduates, and "at-risk" students would graduate 
at a rate of 40%. 

e Including even more predictor variables may yield better indicators. Promising variables include 
course performance and credit accumulation. 
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Exploring Longitudinal Outcomes and Trajectories of English 
Language Learners (ELOTE) 


Project ELOTE is a Researcher-Practitioner Partnership funded by the Institute of Education Sciences.’ 
Researchers from the Quality Teaching for English Learners initiative at WestEd partner with practitioners 
in the Fort Worth Independent School District (FWISD) to investigate academic trajectories and outcomes 
of secondary immigrant newcomers. The partnership describes academic trajectories and identifies 
promising programs and practices for newcomers. This report is part of Phase One, a longitudinal analysis 
of extant administrative data from FWISD. The results from Phase One will enable the identification of 
schools sites for Phase Two, which consists of case studies of promising programs and practices. These two 
phases will directly inform policy and practice within the district. 


English Language Learners in Texas and Fort Worth 


English language learners (ELLs) are a large and growing population within the United States, Texas, and 
Fort Worth. Nationally, ELLs constitute 9.1 percent of the K-12 public school population, or 4.4 million 
students in the 2011-2012 school year (Kena et al., 2014). Texas enrolled approximately 809,000 ELL students 
in the 2013-14 school year, or 16.5% of all K-12 public school students (Texas Education Agency, 2015). 
Within the Fort Worth Independent School District (FWISD), the ELL population has grown both 
numerically and as a fraction of the total school population, from 21,300 in 2003-2004 to 25,800 in 2013-2014, 
corresponding to an increase from 26.6% to 30.6% of the district population. Ensuring that the unique 
educational needs of ELLs are met is an urgent priority. 


Although the majority of ELLs at the national, state, and district levels are concentrated in the elementary 
grades, immigrant newcomers constitute a substantial fraction of the secondary ELL population. In FWISD, 
about 40% of ninth grade ELLs are new immigrants, and about 15% of ninth grade ELLs immigrated during 
the middle school years. Within FWISD, all secondary immigrant newcomer students attend the 
International Newcomer Academy (INA) for one year. At INA, students attend content-area classes taught 
by teachers who work in interdisciplinary teams sharing a group of students. After that initial year at INA, 
students enroll in Language Center programs hosted at four middle schools and seven high school sites for 
one or two years. These Language Centers provide sheltered support in content-area classes for academic 
and English language development. As students make progress in English proficiency, they enroll in 
Transition English as a Second Language (ESL) or mainstream classes. This sequence of programs is 
different from those in other districts across the state and country, which may include mainstream classes 
with various models of ESL support, small programs hosted at comprehensive high schools, two-year 


programs for newcomers, or four-year newcomer high school programs. 


"The research reported here was supported by the Institute of Education Sciences, U. S. Department of 
Education, through Grant R305H140032 to WestEd. The opinions expressed are those of the authors and do 
not represent views of the Institute or the U. S. Department of Education. 
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Predictors and Indicators of Graduation 


This report builds upon a previous report which described the trajectories and outcomes of a cohort of 275 
students who enrolled as first-time ninth graders at the International Newcomer Academy (INA) in the 
2009-2010 school year (Chu & Fong, 2015). By the end of the 2013-2014 school year, these students had either 
graduated (128 students, or 47%) or left the district as non-graduates (147 students, or 53%). The objective 
of this report is to explore variables available in the first year of the study which predict graduation 
outcomes. These predictors could be combined to produce early warning indicators of students at-risk for 
not graduating. 


We distinguish in this report between predictors—which are continuously scaled, like probabilities—and 
indicators, which are binary. Indicators are yes-or-no variables which may be more useful to practitioners 
because they suggest whether or not to target a student for intervention. To have practical validity, an 
indicator ought to correspond to a variable connected at face value to the outcome of interest. Continuous 
predictors can be used to create indicators with empirically set cut-off points, which in turn drive 
programmatic decisions. 


In this report, we are interested in "at risk" indicators for the outcome of non-graduation. Three measures 
need balancing, depending on the priorities and resources of policymakers. The first is the overall accuracy 
(“Accuracy”), which is the percentage of all students for which the indicator and the outcome matched 
perfectly. The Accuracy of an indicator is of particular interest to researchers as it provides a measure of 
how closely the indicator matches the outcome. This perfect matching may be less important in practice 
and making policy. For instance, if an intervention has no negative side effects for participants, perfect 
accuracy may be less important than identifying more students who are at risk. 


Therefore, a second measure is the non-graduate identification rate (“Identification Rate”), or the 
percentage of all non-graduates who are identified as "at risk". This Identification Rate alone is not enough 
because an indicator that identified all students as "at risk” would necessarily identify all non-graduates. 
Accuracy therefore serves as an important check on the overall quality of an indicator. 


The last measure is the "At-risk Graduation Rate”, the percentage of "at risk" students who actually did 
graduate. This "At-risk Graduation" rate is part of the inaccuracy of the indicator, as the student did actually 
graduate even though the indicator identified him or her as "at risk". 


It is desirable for Accuracy and the at-risk Identification Rate to be high. Meanwhile the third measure, the 
At-risk Graduation Rate, ideally should be lower, and a reasonable benchmark for comparison is the 
observed graduation rate for all newcomer students. 


This study contributes to understanding of indicators of graduation outcomes by centering analysis on a 
cohort of secondary immigrant newcomers. Most studies begin with all ELLs in a district and then specify 
predictors to subgroups including newcomers (e.g., Gwynne, Ehrlich, Pareja, & Allenworth, 2012). Other 
studies, which begin with the population at large, find that "on-track" indicators have less predictive power 
for ELLs, and do not report the strength of indicators for high school newcomers as a subcategory (Kemple, 
Segeritz, & Stephenson, 2013). This study specifically seeks to explore whether extant variables can provide 
guidance about how to target newcomers for interventions. 
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Research Questions 


This report answers the following research questions: 


I. To what extent is first-year English language proficiency associated with graduation outcomes? 


II. To what extent do first-year English Reading proficiency and attendance predict graduation 


outcomes? 
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To what extent is first-year English language proficiency 
associated with graduation outcomes? 


Students’ English language proficiency is measured in this report using the Texas English Language 
Proficiency Assessment System (TELPAS). The TELPAS consists of four domains: Speaking, Listening, 
Reading, and Writing. The Writing rating is based upon holistically rated student writing collections, while 
the Speaking and Listening ratings are holistically assessed by ongoing classroom observation. The 
numerical values of the ratings range from 1 to 4 and correspond to the following performance levels: 
Beginning, Intermediate, Advanced, and Advanced High. 


The focus of this section will be the Speaking, Listening, and Writing ratings, because they are ordinal 
variables.” We cross-tabulate first-year ratings in each of these three domains with the dichotomous 
graduation outcome. We then report and compare graduation rates for rating subgroups. We assess the 
significance of associations with the appropriate statistical tests. First-year TELPAS scores were available for 
239 out of 275 students (87%). Among tested students, the graduation rate was 53%, the figure which serves 
as the benchmark for subsequent comparisons in this section. 


Students who were rated Advanced or higher? in Listening were approximately 8 percentage points more 
likely to graduate (61%) than the rest of the cohort (see Figure 1). Students who were rated Beginning in 
Listening were approximately 7% less likely to graduate (46%). The largest Listening subgroup was students 
rated Intermediate, and they were slightly less likely to graduate (51%) than the group at large (see Table 1). 


Figure 1 


Graduation Outcomes by First-Year TELPAS Listening Rating 
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* The TELPAS Reading scores are reported as ordinal performance levels. The Appendix includes an analysis of this 
variable, but we omit it from this section because we analyze Reading as a continuous variable in the next section. 
> To avoid expected values of less than 5 per cell, we combined the Advanced High level with the Advanced level to 
form the "Advanced or higher" category. 
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Table 1 


Ratings Levels and Graduation Rates for First-Year TELPAS Listening, Speaking, and Writing 


Listening Speaking Writing 
Number of Graduation Number of Graduation Number of Graduation 
Rating Level Students Rate Students Rate Students Rate 
Beginning 63 46% 84 46% 68 41% 
Intermediate 107 51% 107 54% 131 55% 
Advanced or higher 69 61% 48 60% 40 65% 


In terms of Speaking, more students were rated as Beginning (84 students, or 35%) as compared to the 
Listening domain, but the Beginning students on Speaking were about as likely to graduate (46%) as those 
rated Beginning in Listening. Compared to Listening, fewer students were rated as Advanced or higher in 
Speaking (48 students, or 20%), and these students were approximately 7% more likely to graduate (60%) 
than the group at large (see Figure 2). 


Figure 2 
Graduation Outcomes by First-Year TELPAS Speaking Rating 
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Figure 3 


Graduation Outcomes by First-Year TELPAS Writing Rating 
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Students rated as Advanced or higher on Writing were the most likely to graduate (65%), approximately 12 
percentage points more likely than the average of all tested students (see Figure 3). Conversely, students 
rated as Beginning in Writing were much less likely (41%) to graduate, the lowest of any subgroup. Writing 
ratings were the most strongly associated with graduation outcomes.* 


* To determine whether these associations were statistically significant, we performed a chi-squared test for 
independence between ratings levels and graduation outcomes. Graduation outcomes were not significantly 
associated with first-year Listening ratings (y?(2)=3.0437, p=0.2183 ) or Speaking ratings (y?(2)=2.5694, 
p=0.2767). The null hypothesis that first-year Writing ratings and graduation outcomes were independent 
was rejected at the 5 percent significance level (42(2)=6.3193, p=0.04244). 
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Predicting Graduation Outcomes with Listening, Speaking, and Writing Ratings 


Because most students (45% to 55%) were Intermediate and the graduation rates of the Intermediate 
students were close to the aggregate rate of 53% (51% to 55%), these ratings do not produce high-accuracy 
indicators of the graduation outcome. Some indicators, however, may be sufficient depending on the nature 
and the scope of the intervention. 


For instance, to construct an "at risk" indicator for non-graduation based upon the domain with the 
strongest association, we could classify all students who rated as Beginning in Writing as "at risk" of 
dropping out. This indicator would identify 68 students, 40 of whom ended up as non-graduates. The 
overall accuracy of this indicator would be 58% (it correctly identified 40 non-graduates as "at risk" and 98 
graduates correctly as not "at risk", a total of 138 out of 238). This indicator has an Identification Rate of 35% 
of all non-graduates (40 out of 113). On the other hand, 41% of the students the indicator identifies as "at 
risk" actually graduate (28 out of 68). Overall, while this indicator does identify students approximately 12% 
less likely to graduate, it only identifies over a third of students who end up not graduating. This indicator 
might be appropriate for an expensive intervention which could not be offered to many students. Any 
improvements to the graduation rate, however, would have to be compared against the baseline graduation 
rate of 41% within the group of identified students. 


If, on the other hand, the objective was to identify more students who end up not graduating, students who 
were Intermediate in Writing may also be identified as "at risk". This inclusion improves some measures but 
moves other measures in the wrong direction, as Intermediate students in Writing are actually no less likely 
to graduate than average. This choice would identify a total of 199 students as "at risk", of whom gg actually 
dropped out. The overall accuracy of this indicator is 52% (correctly identifying 99 non-graduates and 26 
graduates, or 125 out of 238). This indicator has an Identification Rate of 87% of all non-graduates (99 out of 
113), and an At Risk Graduation Rate of 50% (100 out of 199). Although this expanded indicator has a much 
higher Identification Rate (identifying five out of six eventual non-graduates), the students it identifies on 
average graduate at a rate (50%) very close to that of all tested students. Because many more students 
would be identified as "at risk", the delivery of the intervention would be most costly. 


These considerations point at the need to find some way of identify students who are "at risk" within the 
Intermediate level in terms of English language proficiency. To do so, we will use a continuous predictor 
variable in the next section. 
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Il. To what extent do first-year English Reading proficiency and 
attendance predict graduation outcomes? 


This section considers the extent to which two first-year variables—English Reading proficiency and 
attendance-predict graduation outcomes. The TELPAS Reading score is a continuously scaled variable 
ranging from 340 to 950°. The attendance variable is measured as the number of days absent in the first 
school year, and ranged from 1 to 65. Data was missing to different degrees: 243 students had a Reading 
score, while 269 students had attendance data. 


Reading Proficiency and Attendance as Predictors 


For both variables, we employed separate logistic regression models,” with the results showing how the 
probability of graduation changes as the predictor varies. These probabilities are easier for practitioners to 
interpret than raw values of variables such as Reading scale scores, and provide more information than 
discrete categories such as the TELPAS performance levels, which may combine groups of students with 
different proficiency levels and therefore needs. 


Figure 4 
Predicted Probability of Graduation by TELPAS Reading Score 
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First-year TELPAS Reading Score 


For example, this graph shows that a student scoring 650 on the TELPAS reading in the first year has a 
predicted probability of graduation of approximately 50% (about the aggregate average). For a student 
scoring at the lower end of Advanced, a scale score of 680, the predicted probability of graduation is 56%. 
A student scoring above 740 (Advanced High) has at least a 67% predicted probability of graduation. 


° In the 2009-2010 administration, a TELPAS score below 633 was considered Beginning, Intermediate was 634-680, 
Advanced was 681-737, and any score 738 or above was Advanced High. 

° The equation in each case was In(p/(1-p))=Bo+B 1X+. x was either the TELPAS Reading Scale Score or the Number 
of Days Absent. 
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Figure 5 


Predicted Probability of Graduation by Number of Days Absent 
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The predicted probabilities of graduation based upon days absent has two notable features (Figure 5). First, 
the curve runs in the opposite direction from Figure 4, meaning that predicted probability declines as 
absences increase. The shape of this graph makes sense as students with more absences in the freshman 
year are less likely to graduate. Second, the predicted probability of graduation given no absences is 
somewhat low at 58%. Although this value of this predicted probability is low, it is worth comparing to the 
predicted probability for a student scoring 680 on the TELPAS Reading, which was 56%. Reporting 
probabilities thus provides a means to compare variables as different as attendance and Reading test scores. 
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From Predictors to Indicators 


The analyses in the previous section enable modeling probabilities from the two variables. To create 
indicators of student who are “at risk" of not graduating requires setting cut-off values. These values can be 
set using the probabilities derived from the logistic regression, or by directly using the variables. In this 
section, we first model using the raw values of reading proficiency. We use raw values because they are the 
most practically accessible to and readily interpretable by practitioners. At the end of this section, when we 
combine the two indicators, however, we will use probabilities, as that common scale enables us to 
combine test scores with absences. 


Figure 6 


Indicator Performance by Cut-Off Position, TELPAS Reading Scale Score 
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Setting the level for an at-risk indicator based upon the TELPAS Reading Scale Score requires some 
trade-offs between accuracy, identification, and mis-identifying students as at-risk who actually will 
graduate (Figure 6). For example, using 650 as a cut-off (meaning everyone who scores below a 650 is "at 
risk") will identify 57% of all students who actually do not graduate ("Identification Rate"). Approximately 
40% of the students who are identified as at risk, however, will graduate (“At-Risk Graduation Rate”), and 
the indicator will have an overall accuracy of 60% (“Accuracy Rate”). One reasonable approach is to use the 
observed graduation rate as a cut-off. A TELPAS Reading scale score of 670 corresponds to a 54% percent 
chance of graduation, the graduation rate for students with scores. Using this cut-off would identify 66% of 
all non-graduates, while students identified as "at-risk" would have graduated at a rate of 43%. The overall 
accuracy of this indicator would still be 60%, but there would be an increase in the Identification Rate. 
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Figure 7 


Indicator Performance by Cut-Off Position, Number of Days Absent 
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When considering attendance, students are at risk if they have more than a given number of 
absences(Figure 7). For example, if the cut-off is five days (students with at least five absences are "at risk"), 
approximately 83% of all non-graduates will be identified (“Identification Rate”). The graduation rate of 
those then identified at risk (“At-risk Graduation Rate”), however, is 45% and approximately equal to the 
overall value of 47%. If a higher threshold of ten days absent is selected, less than 60% of non-graduates will 
be identified as at risk ("Identification Rate"), but the graduation rates of those at risk ("At-risk Graduation 
Rate") will be approximately 40%. In this case, the overall accuracy ("Accuracy Rate") is about the same 


(54%). 
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Figure 8 


Indicator Performance by Cut-Off Position, Joint Probability Cut-off 
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One option for constructing indicators with higher accuracy and identification is to combine the TELPAS 
reading and absences data, in the form of probabilities of graduating (Figure 8). A student is at risk if either 
predicted probability is less than the cut-off value. For example, if 0.45’ is used as the cut-off for at-risk, 
approximately 45% of all non-graduates will be identified ("Identification Rate"), but approximately 30% of 
all "at risk" students will end up graduating ("At-risk Graduation Rate"). Alternatively, a probability of 0.50° 
identifies 70% of all non-graduates ("Identification Rate") but 40% of all "at risk" students will end up 
graduating ("At-risk Graduation Rate"). 


These values can be used to make decisions about the scope of interventions in terms of the number of 
students that can be served. Attending to these values will enable policymakers to more efficiently target 
students who are at greater risk of not graduating, while not identifying students who are more likely to 
graduate. 


’ This probability corresponds to more than 14 days absent or a TELPAS Reading score below 620. 
° This probability corresponds to more than 8 days absent or a TELPAS Reading score below 646. 
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Implications 


This analysis of a historical cohort provides some insights into how data available early in a newcomer's 
enrollment in the district is associated with summative outcomes. Because only two continuous variables 
were available, the accuracy of predictors was limited. If more data—such as grade point average, course 
completion, and credit accumulation—were available, the accuracy of predictors could be improved. Other 
studies in other contexts have pointed at the promise of these variables, even though those studies have not 
focused explicitly on the subgroup of newcomers (e.g., Gwynne, et al., 2012). 


As policymakers and practitioners explore interventions to improve newcomer retention and graduation, 
they will have to consider how to best target interventions on newcomers most at-risk of not graduating. 
Because interventions have a financial cost, policymakers will have to balance the rate at which "at risk" 
students are identified with the accuracy of the indicator. Some interventions may be lower cost, in which 
case more students can participate, including students who would have graduated even without the 
intervention. Other interventions may be more limited in scope and may need careful selection of 
participants using multiple criteria beyond the indicators identified in this report. 


This report focused on predictors and indicators based upon data from a student's first year of enrollment 
within the district. Tiered interventions may also be implemented by incorporating more data from later 
years of a student's enrollment, such as attendance or course performance in subsequent years of 
enrollment (Frazelle & Nagel, 2015). Intervention systems may have multiple components which address 
different indicators, with varying degrees of response, ranging from school-wide to targeted to intensive. 
This preliminary analysis can serve as an initial screening to identify students for further attention from 
campus-based staff. 


Future analyses will explore campus-specific patterns and other characteristics of non-graduates. 
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Appendix. First-year TELPAS Reading Rating and Graduation 


Table Al. 
Graduate Status 
Graduation 

Reading Rating No Yes Rate 
Beginning 45 32 42% 
Intermediate 28 40 59% 
Advanced or 
higher 34 54 61% 

107 126 54% 
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We tested the null hypothesis that TELPAS Reading proficiency level was not associated with the 
graduation outcome for the 233 students who had proficiency scores. We rejected the null hypothesis at 
the 0.05 significance level (y2(2)=7.357, p=0.02526). 
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